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A Web-based Delphi process can be used to answer difficult questions, compile a body of knowledge from 
experts, or solve a problem or establish content validity. Because of its more qualitative online discussion 
environment, a Web-based Delphi procedure has the potential to offer a more rigorous validation of HRD- 
related content than traditional paper-based Delphi procedures. The method also improves ethics in 
research by insuring anonymity and confidentiality. 

Keywords: Delphi Technique, Validation, Web-based Research 

1. To what extent can an instrument be developed by a Delphi expert panel to measure the application of adult 
learning principles to fully-mediated World Wide Web-based distance education courses? 

2. To what extent is there consensus among Delphi panel experts in the fields of adult education and Web-based 
course development to validate specific instructional methods and techniques that demonstrate the application 
of adult learning principles to fully-mediated World Wide Web-based distance education courses? 

3. The Delphi method, traditionally a paper-pencil technique can be established as a web-based method to validate 
research measures. 

The above propositions were explored in a recent study that used the Delphi research method to develop the Online 
Adult Learning Inventory (http://www.mpd.edu/sharon_colton), an instrument to apply the principles of adult 
learning to Web-based instruction and training (Colton, 2002). A pioneering feature of this study was conducting the 
Delphi process on the Web rather than employing the traditional paper and pencil or computer network Delphi 
techniques. A Web site was constructed with a threaded discussion forum for discussions related to developing 
content and validity, Web forms for voting purposes to determine the level of expert consensus, a calendar to keep 
the panel on task, and as an archive to hold draft versions of the instrument and the text of previous discussions 
available for review at any time by the expert Delphi panel and researchers. The experts were assigned pennames for 
anonymity within the group. Ample time was allotted for expert panel members to reflect on the content of the draft 
instrument and to add additional commentary to the discussion forum any time and from any place. 

The purpose of this paper is to highlight the results of an online Delphi research project; in particular the 
procedures used to establish an online Delphi and to describe a new process of validating HRD-related content and 
obtaining ‘rich' and descriptive information using the World Wide Web and current e-learning technologies. The 
online Delphi was proven to be an excellent tool in establishing content validity. Historically, Delphi technique is an 
overlooked research method due to its labor-intensive nature. This study illustrated online Delphi as a powerful, 
effective and efficient research tool for HRD and adult education research. 

Most Delphi procedures are paper-based with some previous Delphi’s conducted on a mainframe computer or 
network (Turoff & Hiltz, 1995). The original purposes in constructing the Delphi Research site on the World Wide 
Web was (1) convenience, (2) elimination of paperwork and mailings, and (3) an attempt to utilize current 
technology, a topic of the research, into the research process. The result was the sheer volume of rich discussion and 
the “anytime, anywhere” give-and-take communications amongst panel members leading to eventual consensus and 
content validation. Delphi procedures have had some limited qualitative aspects to the otherwise quantitative voting 
procedures and consensus. This study has demonstrated the potential of the World Wide Web to expand the 
qualitative aspects of the Delphi procedure to a great extent, resulting in a more in-depth content validation. 

Delphi Research Method: A Review of the Literature 

Computer-based Delphi procedures have been used since the 1970s on mainframe computers or networks (Turoff & 
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Hiltz, 1995). Today, however, the technology is available to conduct an anonymous asynchronous threaded 
discussion easily on the Web “...where the merger of the Delphi process and the computer presents a unique 
opportunity for dealing with situations of unusual complexity” (Turoff & Hiltz, 1995 p.9). Research indicates this 
combination opens the possibility for greater performance from the Delphi panel of experts than could be achieved 
from any individual, something that rarely happens in face-to-face groups (Turoff & Hiltz, 1995, p.8, p.l 1). 

Delphi panelists are typically selected, not for demographic representativeness, but for the perceived expertise 
that they can contribute to the topic. In order to obtain the desired valid results, Scheele (1975) suggested the panel 
must be selected from stakeholders who will be directly affected, experts with relevant experience, and facilitators in 
the field under study. Spencer-Cooke (1989) suggested that the composition of the panel relate to the validity of the 
results of the research. 

Delphi panel sizes range from a few to fifty or more participants. In Brockhoff' s study of Delphi performance 
(1975), he suggested that for forecasting questions, groups with eleven participants were more accurate in their 
predictions than larger groups. For fact-finding questions as included in this dissertation, groups with seven 
participants had a higher performance in his controlled study. Other studies have found that error decreases with 
larger Delphi panels (Linstone & Turoff, 1975). Dalkey (1975), one of the originators of the Delphi research 
method, commented that, “. . .under favorable conditions, the group response can be more accurate than any member 
of the group” (p. 257). 

The time requirement for the Delphi process was significant. The process can last for 30 to 45 days (Barnes, 
1987) but in this Web-based study, it took several months. For that reason, Scheele (1975) states that attractive and 
stimulating peers provide the most powerful incentive to participate. Turoff and Hiltz note that, “Motivation for the 
expert is often lacking because results are often delayed or are not intended to benefit the expert” (1995, p.9). The 
participants were offered the opportunity to participate in the discussion with other panel members of equal merit, to 
participate in producing and validating an evaluative knowledge-based tool for others, and to experience a Delphi 
process. It is also necessary for the panelists to be assured that the facilitator (researcher) has an understanding of the 
content. The response rates of the experts for paper-based Delphi method dissertations include, 92% overall with 
less than a ten percent drop-out rate (Dobbins, 1999, Cooter, 1983; DeLap, 1998; Stover, 1997; Jackson, 1998), and 
similar results with the one Web-based Delphi discussed in this article (Colton, 2002). Participants who responded 
slowly or not at all to calls for participation were contacted by telephone or sent additional e-mail reminders in order 
to gain a higher level of participation. 

Delphi procedures normally consist of three or more rounds. Each round consists of answering questions posed 
and is ended by a vote. This typically is conducted with paper and pencil. There is usually a decrease in response 
rates for the second round of a paper-based Delphi, particularly when using volunteers, as they may lose interest 
(Jillson, 1975). The author’s Web-based Delphi study had a similar response rate. 

Computer-based techniques are far better than paper and pencil in constructing a flexible approach and, in fact, 
the traditional round structure may disappear, replaced by a continuous feedback process (Linstone & Turoff, 1975). 
The Delphi method became less structured and permitted greater freedom of discussion after the use of computer 
conferencing began in 1969. Adding the computer capability allowed for a shorter turn-around time, allowing for 
more interaction and more material discussed (Price, 1975). 

The characteristics or benefits of conducting the Delphi process via computer over a face-to-face discussion as 
summarized from Price (1975) include: 

1 . When the communication process must be structured. 

2. When the problem is so broad that many more individuals are needed than can interact face-to-face. 

3. When severe disagreement among participants occurs, the process must be refereed, and anonymity 
must be assured within the group of experts. 

4. When time is scarce and/or geographic distances are great, limiting group meetings. 

5. When an easier more flexible way to access and exchange human experience is required. 

6. When increases of the size of the information space to infinity is desirable. 

7. When raising the probability of developing latent consensus is desirable. 

8. When a written record is desirable. 

There are potential problems in the Delphi process and in the contemporary computer-based method. The 
historical Delphi model follows a sequential, paper-based structure with the facilitator acting to summarize the 
round. The computer discussion method can prolong the procedure and discussion can assume parallel tracks. Turoff 
(1991) suggested the timely use of voting to integrate the problem solving process with the group process. Turloff 
(1991) summarizes the use of computers in the process as, “The merger of Delphi and Computer Mediated 
Communications offers far more than the sum of the two methods” (p. 11), by which he implies that by introducing 
computer-based discussion into the Delphi process to replace paper and pen, the Delphi process was strengthened. 



9-1 




185 



Turoff, in recommending using the Internet for discussion, emphasizes that the most important criterion to Delphi 
process design is allowing any panel member to “choose the sequence in which to examine and contribute to the 
problem solving process” (p. 2). 

There are many reasons for the use of anonymity in the Delphi process. The reasons include: an expert making a 
commitment to a stand then being reluctant to change it, the different academic standings of the participants, not 
losing face, and elimination of the usual biases found in today’s society such as gender, racial, and age biases. 
Anonymity of responders allows consensus to take place without the undue influence of rank, power, personality or 
persuasive speaking which is common to group meetings (Westbrook, 1997). Hiltz, Turoff, and Johnson (1989) 
suggest the use of pen names when using computer-based communications. A forum or electronic bulletin board 
enables this technique as e-mail addresses are not used for communications, thus anonymity can be assured amongst 
the panel members. It is essential that the researcher is the only one who can connect the pen name with the panel 
member. Pen names, although anonymous, allow for a sense of identification within the community of experts and 
were used for the present study (Colton, 2002). 

Since the results of a Delphi are produced by structured interaction, the final product can be said to constitute a 
“reality construct for the group” (Scheele, 1975, p. 44). The results of a Delphi can be seen as “the product of a 
carefully designed and managed interaction and not answers to a set of abstract questions that are obtained by 
following prescribed methods” (p. 38). 

“People incorporate each others’ perspective and information into their thinking and arrive at a fairly accurate 
understanding of the critical issues to consider in their decision making process” (M. T. Corporation, 1983). 
Panelists may change their previous votes at any time (Turoff & Hiltz, 1995). If consensus was not achieved on an 
item, that item may be dismissed for the present, subject to a later revision. Brockhoff (1975) states that variance 
reduction, or consensus, almost always occurs in Delphi groups between the first and fifth rounds but the best 
results, as a rule, are already known by the third round. Thus, any additional discussion may not be necessary. 

Web-base Delphi Procedures 

The present study used a web based Delphi method to develop and establish content validity of the Online Adult 
Learning Inventory. Research methods for validity included: (a) a thorough review of the literature to construct an 
item pool of instructional methods and (b) Delphi expert panel consensus. The mean, mode, standard deviation, 
interquartile range, and skewness of the data were calculated from the voting procedures for determination of 
consensus. Evidence of reliability was indicated by the interrater reliability coefficient from a field test. In addition, 
an informal review of readability was conducted to improve the readability of the instrument and the Gunning Fog 
Index (1983) for readability was calculated. 

The pioneering Web-based Delphi process proved to be a method rich in qualitative data and was an excellent 
way of bringing together experts to discuss, debate, and organize a body of information in order to develop a 
validated instrument, reach agreement on an issue, uncover common factors, or forecast trends. This method has 
potential for use by researchers to build a validated knowledge construct utilizing the resources of the World Wide 
Web for convenience, ease of use, and depth of discussion. Business and industry personnel may use the Web-based 
Delphi method to validate instruments or knowledge constructs. 

Specific information on constructing a Delphi Web site can be found on Website: 
http://www.mpc/sharon_colton. The following is a recommended procedure for online Delphi to develop and 
establish content validity of an instrument. A visual representation of the procedures used in the present study is 
illustrated in Figure 1. Web-based Delphi procedures: 

1. Literature review: Preliminary content collected for the instrument using established quality filters, 
criteria for selecting the expert panel established, and appropriate and established research methods are 
selected. In the present study the principles of adult learning were reviewed, as were web-based 
instructional methods. 

2. Selection of the expert panel : Selection criteria for panel members must be based on a review of the 
literature, potential panel members are then selected based on the criteria, and approval of the potential 
expert panel members was obtained from the studies’ sponsor(s). Human subjects protections should 
also be established at this time as applicable. Potential panel members are reached by telephone to seek 
their acceptance. Upon acceptance, a follow-up letter along with any required release forms is faxed or 
e-mailed to the experts who accepted the invitation to participate. See Table A, Procedure for selection 
of expert panel members. 
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Figure 1. Web-based Delphi Procedures 
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Table A. Procedure for Selection of Expert Panel Members 



Step 


Procedure 


Result 


Step 1 


Review literature to compile a list of potential panel 
members based on one recent (since 1995) book or journal 
article on Web course development. 


Compile list of names. 


Step 2 


Check books or articles (or other articles or books by the 
same author) for evidence of knowledge of adult learning 
principles. 


Mark for evidence of adult learning 
principles. 


Step 4 


Check ISI Social Sciences Citation Index for number of 
citations. 


Mark number of citations. 


Step 5 


Evaluate potential experts as to their contributions to the 
scholarly discussion of adult learning, courseware 
development, or familiarity with instructional methods 
appropriate for Web courses. 


Rate potential experts on a 
suitability-to-the-study scale of 1 to 
3(1= not useful, 2 = moderately 
useful, 3 = very useful to the study). 


Step 6 


Present evidence of potential panel member’s expertise to 
the Dissertation Committee members for review. 


Develop a final list of potential 
expert panel members to invite to 
participate and a list of substitutes. 


Step 7 


Telephone each potential panel member to explain the 
purpose and scope of the study, with invitation to participate. 


Follow-up w / ea participant 

committed to the study with letter. 



3. Review of readability: Knowledgeable subject matter experts review the preliminary draft instrument 
for appropriate wording and ease of understanding. Revision is made to wording based on suggestions. 

4. Set-up of the discussion forum: The discussion forum is set up on the Web site with the latest revision 
of the instrument and other data attached to the website. Pen names and passwords are selected for 
each participant to insure anonymity. Choose gender-neutral pen names for participants. 

5. Round one of the Delphi : Establishment of instrument content by discussion and vote for possible 
consensus. In the present study the experts were given a draft instrument with adult learning principles, 
as derived from the literature, as the structure of the instrument. The main points of consideration 
were: Is the principle relevant to web-based course development, and, if so, is it worded correctly? 
They had three weeks to discuss items on this list, suggest changes to the list, collapse any two 
principles into one, separate one complex principle into two separate principles, alter wording and 
phrasing, and make additional comments that came to mind. This phase of the Delphi resulted in a 
depth of discussion not previously obtainable in paper-pencil based Delphi procedures. For example. 
Mango (penname) asked to soften the language, especially the wording “to cope effectively” as it 
suggests survival, yet many adults enroll in courses for pleasure. Mango’s suggestion was to re-word 
the principle as: “Adults become ready to learn those things ... with which they can or wish to relate 
their real-life experiences.” Celery (penname) agreed that the principle needed to be re-worded and 
refers to the “primacy” in adult learning, the need to know something becomes of primary importance. 
Although agreeing with the above experts. Broccoli (penname) stated that adults are motivated by life 
events “to pursue formal educational opportunities and to conduct informal learning projects.” Tomato 
(penname) related that, “we design instruction to solve a problem.” Kiwi (penname) offered, “Web- 
based learning systems can and should connect learning to the learner's life experiences” as an 
alternative re-wording. Kiwi commented that “the concept, premise, principle applies to all learners, 
but might be more relevant or pertinent to adult learners.” 

Experts then had another two weeks to vote on the list. Prior to voting, the list of adult learning 
principles was revised based on suggestions by the expert panel. Voting ended the round. Results of 
round one were displayed on the discussion forum. Mean, median, mode, standard deviation, and 
interquartile range were calculated. Based on the suggestions and a statistical analysis of the vote, the 
instrument and its structure of adult learning principles were revised again. 

6. Round two of the Delphi : Round two consists of establishing and sorting of an item pool completed by 
a vote. Consensus is typically not expected at this point. Expert panel members in the present study 
were asked to list one or more instructional methods that applied an agreed-upon adult learning 
principle to Web instruction or training for adults. Because of the opportunity for discussion and 
debate that a threaded discussion forum afforded, there was expected to be some negotiation toward 
consensus during the dialogue. Results of the listing of instructional methods were displayed on the 



9-1 





188 



discussion forum. One week was given to the expert panel for reflection on the draft instrument as 
again revised with the list of instructional methods included. Then, a vote was conducted on the large 
item pool or list of instructional methods, which applied the various adult learning principles to Web 
courses, using a Likert scale. Descriptive statistics were calculated to indicate consensus. Edits were 
made by the researchers to the list of instructional methods based on the results of the vote, comments 
on the voting ballot, correspondence, and references from the literature where necessary. Items 
receiving weak consensus (mean of 3.0 or higher and an interquartile range of 2 or greater) were 
retained for a re-vote for the third round to allow panel members to consider changing their vote. 

7. Round three of the Delphi : Follow up discussion was available and a second vote was performed on 
the revised list of instructional items either to include in the instrument or consider for elimination. 
Statistics were calculated as before. Items not having reached consensus to be included in the 
instrument were considered for elimination from the final instrument. Edits were made to the list of 
instructional methods based on the results of the vote, comments on the voting ballot, correspondence, 
and references from the literature where necessary. 

8. Field test for indication of reliability. A field test was conducted using fourteen university or 
community college faculty or staff who had knowledge of Web course development and/or evaluation. 
Participants first participated in an overview of adult learning principles, then used the draft instrument 
to conduct an evaluation of one pre-chosen instructional Web course. Comments by the participants 
related to the draft instrument were recorded. Results of the Web courses review were analyzed for an 
indication of inter-rater reliability using standard correlation procedures for estimating agreement 
corrected for chance. The inter-rater reliability statistic gives an indication of the reliability and 
consistency of the instrument. Participant comments and results of the analysis were used for the final 
revisions of the instrument. The Gunning FOG Index (1983) was also computed for final reading level. 

Conclusions and Implications for HRD Research and Practice 

The Web-based Delphi process used for this study is new to the field of research design. This study demonstrated 
the power of technology in enhancing a classic and ethical Delphi research process, in facilitating discussion among 
participants separated by time and place, and providing a venue for voting, all while preserving the anonymity of the 
participants. It yielded rich qualitative and rigorous quantitative data resulting in a content validated instrument, 
possibly resulting in a more in-depth content validation, applicable to educational, business, industrial, and 
government research as well as bringing the tenets of andragogy into the 2 1 sl century. The procedures used in this 
study can also be applicable to HRD practice in that with the ubiquitous nature of the technologies used in the study 
it makes replication of the process feasible for practitioners and others seeking to develop valid tools to measure 
HRD and adult education related content. 

Finally, this process insured anonymity of participants and confidentiality of data throughout the process as well 
as addressing issues around power and rank inherent in expert-based research. Anonymity of responders allowed 
consensus to take place without the undue influence of rank, power, personality or persuasive speaking which is 
common to group meetings (Westbrook, 1997). Required anonymity in the Delphi process is based on: an expert 
making a commitment to a stand then being reluctant to change it, the different academic and reputation standings of 
the participants, not losing face, and elimination of the usual biases found in today’s society such as gender, racial, 
and age biases. 
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